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Abstract 

The  United  States  military  is  performing  operations  in  urban  environments  with 
increasing  frequency.  Current  Department  of  Defense  doctrine  is  poorly  suited  to  train 
and  equip  today’s  warriors  with  the  tools  and  experience  necessary  to  fight  and  win  in 
modem  sprawling  cities.  In  order  to  “close  the  gap,”  the  U.S.  Joint  Forces  Command’s 
Joint  Experimentation  Directorate  led  an  effort  to  run  a  massively  distributed  simulation 
of  a  synthetic  urban  environment  utilizing  human-in-the-loop  operators  called  URBAN 
RESOLVE.  The  synthetic  environment  simulated  the  city  of  Jakarta  with  over  1,000,000 
buildings  and  structures  and  over  120,000  civilian  entities.  A  Red  force  retreated  into  the 
city  while  a  Blue  force  attempted  to  determine  the  enemy’s  Order  of  Battle.  The  exercise 
generated  over  3.7  terabytes  of  data  in  seven  distinct  trials.  This  research  evaluated  the 
time  required  to  identify  targets  after  detection  and  the  accumulation  of  identifications 
over  time,  and  searched  for  trends  between  the  seven  design  trials  and  between  target 
groups.  Two  trends  emerged  from  this  research.  First,  there  was  a  notable  difference  in 
the  time  required  to  identify  a  target  once  it  has  been  detected  based  on  its  target  group. 
Second,  two  design  trials  that  are  expected  to  demonstrate  show  counter-intuitive  results. 
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MODELING  URBAN  WARFARE:  JOINT  SEMI-AUTOMATED  FORCES  IN  URBAN 

RESOLVE 


I.  Introduction 


Background 

In  the  history  of  warfare,  one  of  the  enduring  truths  has  always  been,  attack  cities 
as  a  last  resort.  In  The  Art  of  War  Sun  Tzu  wrote,  “The  worst  policy  is  to  attack  cities. 
Attack  cities  only  when  there  is  no  alternative. . .”  (Tzu,  1971 :78-79).  Among  other 
difficulties  and  complications,  Sun  Tzu  recognized  the  extreme  difficulty  of  attacking 
walled  cities.  Successfully  laying  siege  to  such  a  city,  while  possible,  required  expending 
vast  amounts  of  materiel  and  human  resources.  There  are  few  walled  cities  left  in  the 
world  today,  and  the  lethality  of  modem  weapons  eliminates  their  utility  as  an  effective 
defense  against  attack.  Nevertheless,  modem  urban  combat  presents  its  own  host  of 
difficulties.  Wholesale  bombing  of  cities  and  the  ensuing  civilian  casualties  is  both 
unnecessary  and  politically  unacceptable.  What  is  more  is  the  realization  that  it  is  no 
longer  enough  for  the  United  States  to  defeat  its  enemies;  we  must  make  all  efforts  to 
provide  stability  and  establish  a  culture  of  democracy.  The  result  is  that  the  United  States 
finds  itself  not  only  in  the  unenviable  position  of  engaging  in  open  urban  combat 
operations,  but  also  perfonning  peace  keeping  and  stability  operations  against  an 
insurgency  bent  on  hastening  our  departure. 


1 


The  challenge  facing  the  Department  of  Defense  (DoD)  is  developing  the  tactics, 
training,  and  equipment  that  will  prepare  our  forces  to  fight  in  such  urban  scenarios. 
Before  the  end  of  the  cold  war,  the  DoD’s  primary  focus  was  to  prepare  for  a  massive 
force  on  force  battle  against  the  USSR.  Although  urban  combat  was  sure  to  take  place  as 
the  battle  raged  across  the  European  continent,  little  attention  was  given  to  the  needs  of 
effective  urban  combat.  What  little  attention  there  was,  focused  primarily  on  small 
villages  and  full  force  combat  not  peace  keeping  and  stability  operations  in  a  sprawling 
metropolis  with  millions  of  people,  and  millions  of  buildings  and  structures  (Fontenot, 
2003).  However,  recent  advances  in  computer  technology,  experimental  architecture 
design,  networking,  and  a  renewed  focus  in  light  of  events  in  the  Middle  East,  has 
resulted  in  the  genesis  of  large  scale  distributed  synthetic  environments  designed  to 
provide  the  command  and  control  warrior  the  ability  to  practice  operations  in  a  modern 
urban  landscape. 

URBAN  RESOLVE  is  a  three-phase  exercise  sponsored  by  U.S.  Joint  Forces 
Command’s  (JFCOM)  Joint  Experimentation  directorate  (J9).  The  goal  of  URBAN 
RESOLVE  is  to  “. . .  guide  the  development  of  critical  warfighting  capabilities  for  the 
future  joint  force  commander,  with  a  particular  focus  on  those  needed  for  effective  urban 
operations,”  (Urban,  2006).  At  its  essence,  URBAN  RESOLVE  provides  a  mechanism  to 
evaluate  the  utility  of  near-future  intelligence,  surveillance,  and  reconnaissance  (ISR) 
assets  within  the  urban  environment  and  to  develop  tactics,  techniques,  and  procedures 
(TTP)  to  aid  the  joint  forces  commander  in  gaining  situational  awareness  of  enemy 
activities. 
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URBAN  RESOLVE  utilizes  the  Joint  Semi-Automated  Forces  synthetic 
environment  to  allow  real-time  human-in-the-loop  (HITL)  interaction  by  both  a  Blue  and 
Red  force  within  a  synthetic  version  of  the  Indonesian  capital  of  Jakarta  complete  with 
over  1,000,000  buildings  and  over  120,000  non-combatants  moving  about  the  city  in  a 
culturally  accurate  manner  (Wielhouwer,  2005: 12-13  and  Rafuse,  2006).  URBAN 
RESOLVE  participants  control  a  constellation  of  approximately  250  ISR  assets  that 
generate  tracks  from  approximately  1100  enemy  targets  attempting  to  hide  within  the 
city’s  civilian  population. 

A  critical  aspect  of  such  a  large  simulation  is  the  massive  data  collection  and  data 
analysis  needs.  A  single  twenty- four  hour  run  of  URBAN  RESOLVE  generated 
approximately  100  gigabytes  of  data  and  over  forty  million  rows  of  data.  The  entire  first 
phase,  which  consisted  of  seven  experimental  designs,  generated  over  3.5  terabytes  of 
storage  space.  Yet  despite  all  this  data  there  was  only  a  single  replication  for  each 
experimental  design.  The  lack  of  replications  presents  difficulties  when  perfonning 
statistical  analysis  due  to  the  inability  to  make  assumptions  about  the  data  such  as 
normality  and  independence. 

Problem  Statement 

This  thesis  analyzes  data  produced  from  the  exercise  URBAN  RESOLVE  Phase  I 
by  manipulating  the  data  into  various  formats  and  using  nonparametric  statistics  in  order 
to  find  trends  in  the  data  that  will  provide  additional  insight  into  the  results  of  the 
exercise.  The  challenge  for  this  research  effort  is  to  analyze  a  dataset  containing  over  40 
million  rows  of  data  totaling  3.7  terabytes  in  size.  In  addition,  there  are  only  single 
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replications  for  each  of  the  seven  experimental  design  trials.  Finally,  the  experimental 
design  varied  multiple  design  parameters  between  trials  and  made  it  difficult  to  determine 
the  exact  cause  of  changes  observed  from  trial  to  trial. 

Scope 

This  research  effort  will  focus  on  the  track-data  collected  during  URBAN 
RESOLVE  Phase  I.  The  data  consists  of  the  complete  listing  of  all  tracks  generated 
when  a  Blue  force  sensor  detects  a  target  within  the  simulation.  The  data  includes  target 
type,  detecting  sensor,  and  detection  time.  If  identification  was  made,  the  data  also 
includes  the  identifying  sensor  and  time  of  identification.  Analysis  will  focus  specifically 
on  the  time  required  to  identify  a  target  (TTI),  after  a  detection  has  occurred,  and  how 
that  time  varies  between  design  trials,  target  groups  (soldiers,  civilians,  mechanized,  etc), 
and  the  detecting  platform.  This  research  will  also  investigate  the  rate  at  which 
identifications  are  generated  for  each  trial  and  target  group.  The  intent  is  to  search  for 
trends  in  the  data  that  have  not  been  previously  identified,  and  which  may  indicate  areas 
for  further  study. 

Outline 

The  literature  review  chapter  provides  an  overview  of  modeling  and  simulation  in 
the  Department  of  Defense  (DoD),  a  look  at  combat  modeling  past  and  present,  the 
growing  importance  of  urban  combat,  virtual  simulation,  synthetic  environments,  and 
finally  URBAN  RESOLVE.  The  methodology  chapter  will  discuss  the  URBAN 
RESOLVE  experimental  architecture,  experiment  design,  data  collection  and  reduction, 
and  finally  data  analysis.  Chapter  IV  will  present  the  results  of  our  data  analysis  and 
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Chapter  V  will  provide  a  synopsis  of  the  analysis,  obstacles  encountered,  the  significance 
of  the  research  and  finally  recommendations  for  future  research. 
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II.  Literature  Review 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  provide  an  overview  of  the  various  aspects  of 
combat  modeling  in  a  real-time  high-resolution  synthetic  environment.  Particular 
attention  will  be  paid  to  combat  modeling’s  ability  to  aid  in  understanding  the  complex 
urban  battlefield.  This  chapter  begins  with  a  general  discussion  of  modeling  and 
simulation  and  their  applications  within  the  DoD.  This  is  followed  by  an  overview  of 
attrition  based  combat  modeling  and  its  limits.  Next  will  be  a  discussion  of  the  increased 
role  of  urban  combat  and  asymmetrical  conflicts  faced  by  the  U.S.  military.  Following 
this  is  a  discussion  of  synthetic  environments  and  the  challenges  faced  in  creating 
applicable  synthetic  urban  environments  and  modeling  asymmetrical  threat  behavior. 
Finally,  this  chapter  will  discuss  the  Joint  Forces  Command  sponsored  exercise  URBAN 
RESOLVE. 

Modeling  and  Simulation 

According  to  the  Defense  Modeling  and  Simulation  Office  (DMSO),  modeling 
and  simulation  in  the  DoD  is  broken  into  three  functional  areas:  training,  analysis,  and 
acquisition.  Models  can  be  further  categorized  based  on  their  resolution  and  level  of 
aggregation.  DoD5000.59-M,  Department  of  Defense  Modeling  and  Simulation  (M  &  S) 
Glossary  (1998),  defines  resolution  and  aggregation  as,  “The  degree  of  detail  and 
precision  used  in  the  representation  of  real  world  aspects  in  a  model  or  simulation,”  and 
“The  ability  to  group  entities  while  preserving  the  effects  of  entity  behavior  and 
interaction  while  grouped,”  respectively.  The  DoD  Services  each  have  extensive  libraries 
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of  models  and  simulations  they  use  to  satisfy  these  various  functional  areas.  These 
models  span  the  spectrum  from  modeling  airfoil  fluid  dynamics,  to  operational  planning 
of  a  major  theater  war.  Functionally,  they  range  from  training  operators  in  weapon 
system  simulators,  analyzing  the  effects  of  integrating  a  new  weapon  system  into  the 
inventory,  and  supporting  the  effectiveness  and  suitability  determination  of  an  acquisition 
program.  The  DMSO  Modeling  and  Simulation  Resource  Repository  (MSRR)  contains 
hundreds  of  government  accepted  models  and  simulations  that  may  be  used  for  analysis 
within  the  DoD.  The  hierarchy  and  functional  areas  of  models  and  simulations  used  by 
the  DoD  are  represented  in  Figure  1. 


Additional  M&S  Dimensions 

•  Level  of  Resolution 

•  Degree  of  Human  Participation 

•  Degree  of  Physical  Realism 


Sponsoring  Component 


Figure  1.  Hierarchy  and  Functional  Areas  of  DoD  Combat  Models  (DoD 


5000.59-P,  1995) 
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Combat  Modeling 

Combat  modeling  has  historically  focused  on  modeling  the  attrition  of  forces  in 
open  combat.  Effectively  modeling  a  complex  event  such  as  combat  typically  requires 
some  minimal  level  of  aggregation.  Even  with  the  aid  of  computers,  aggregation  is  often 
required  for  a  timely  answer  due  to  limits  of  processing  speed  and  storage  requirements. 
The  work  of  Frederick  W.  Lanchester  has  dominated  combat  modeling  attrition  analysis 
since  World  War  I.  In  1914,  Lanchester  described  warfare  using  two  basic  models.  The 
first,  based  on  “ancient”  warfare,  assumes  a  battle  to  be  a  collection  of  one-on-one  duels 
and  results  in  a  casualty  exchange  ratio  proportional  to  force  size.  The  second  model, 
based  on  “modem”  warfare,  attempts  to  capture  the  effects  of  weapon  lethality  and  the 
ability  to  concentrate  fire,  which  can  result  in  a  many-on-one  scenario.  Here  the  casualty 
exchange  ratio  is  inversely  proportional  to  the  size  of  the  enemy  force.  These  two  models 
are  termed  the  Linear  and  Square  laws  respectively  (Hartman,  1997:6-2) 

The  Lanchester  attrition  models  aggregate  warfare.  In  doing  so,  they  treat  all 
combatants  identically,  and  ignore  differences  in  individual  combatants  such  as  skill, 
state  of  alertness,  equipment  state  of  repair,  lethality,  logistics,  complex  terrain, 
intelligence  etc.  Over  the  years,  manipulations  of  the  basic  equations  have  allowed  for 
heterogeneous  applications  of  the  Lanchester  models  that  attempt  to  mitigate  some  of  the 
above  limitations.  However,  none  of  the  various  additions  or  complications  are  able  to 
capture  the  impact  of  maneuver  warfare,  information,  command  and  control  (C  ),  civilian 
casualty  avoidance,  and  asymmetric  warfare.  When  modeling  at  the  theater  level  of  war 
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in  a  force-on-force  scenario,  these  facts  may  be  of  little  consequence.  However,  modem 
warfare  is  increasingly  dependant  on  C",  intelligence,  and  maneuver. 

Urban  Combat 

The  battlefield  of  modern  warfare  is  changing  as  well.  In  the  post-cold  war  era, 
the  United  States  has  demonstrated  its  ability  to  prosecute  a  conventional  force-on-force 
battle  with  relative  ease.  Unfortunately,  our  adversaries  have  noticed  this  too  and  avoid 
open  combat  whenever  possible.  In  addition,  US  deployments  abroad  have  grown  in 
number  as  the  world  has  become  more  urbanized  and  the  US  finds  itself  performing 
combat  operations  in  the  urban  environment  with  increasing  regularity  (Wielhouwer, 
2005:2-5).  In  the  US  Army’s  analysis  of  Operation  Iraqi  Freedom,  On  Point  (2003),  the 
authors  discuss  the  fact  that  the  1991  Gulf  War  showed  the  Iraqi  military  the  futility  of 
facing  coalition  forces  in  open  desert  combat.  Their  reaction  to  this  stark  reality  became 
apparent  in  2003  as  they  massed  their  forces  in  urban  areas  such  as  Baghdad.  Although 
victory  was  swift,  the  on  going  insurgency  shows  that  the  urban  environment  is  where  the 
US  is  likely  to  face  its  greatest  future  challenge.  Finally,  US  military  operations  will 
continue  to  include  military  operations  other  than  war  (MOOTW)  in  the  urban 
environment  with  ever-increasing  frequency  (Wielhouwer,  2005:2). 

An  important  acknowledgment  made  in  On  Point  (2003),  is  the  DoD’s 
critical  weakness  at  developing  effective  joint  doctrine,  tactics,  and  training  for  the 
modem  urban  environment.  Until  recently,  urban  combat  training  facilities  were  limited 
to  simulating  small  towns  and  villages,  rather  than  the  vast  urban  metropolises  common 
today.  Furthermore,  DoD  modeling  and  simulation  capabilities  have  lacked  the  detail, 
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scale,  and  real-time  capacity  required  to  realistically  train  in  a  virtual  simulated  urban 
environment. 

Virtual  and  Constructive  Simulation 

The  DoD  must  therefore  construct  robust  simulated  environments  that  will  allow 
military  planners  to  effectively  develop  and  test  doctrine  for  joint  urban  operations 
(JUO).  Virtual  and  constructive  simulations  are  defined,  respectively,  by  DMSO  as  . . 
real  people  operating  simulated  systems,”  and  simulated  people  operating  simulated 
systems.”  A  third  type  of  simulation,  live  simulation,  involves  “real  people  operating  real 
systems.”  (DoD  5000. 59-P,  A-6: 1 998)  Unfortunately,  operating  military  hardware 
within  a  real  urban  environment  is  problematic  at  best.  Nor  are  we  likely  to  build  a  full- 
scale  model  of  a  modern  urban  area  complete  with  non-combatants  going  about  their 
daily,  simulated,  lives.  We  are  therefore  limited  to  training  within  a  virtual  and 
constructive  world  where  key  players  can  develop  and  practice  tactics,  techniques,  and 
procedures  (TTP)  in  a  realistic  simulated  urban  environment. 

A  synthetic  urban  environment  must  include,  at  a  minimum:  a  robust  building  set 
and  terrain  map,  non-combatants  acting  in  a  culturally  accurate  manner,  intelligent 
adversaries,  vehicle  traffic,  and  intelligence,  surveillance,  and  reconnaissance  (ISR) 
assets  (Dehncke,  2005).  These  features,  and  more,  must  be  implemented  so  that  they 
operate  in  real-time  and  allow  for  human-in- the-lo op  (HITL)  interaction. 

Creating  a  complex  urban  environment  is  not  a  simple  task.  Only  a  few  terrain 
maps  exist  with  high  levels  of  detail  and  even  fewer  building  data  sets  exist  that 
realistically  reproduce  the  millions  of  structures  that  may  be  present  in  a  modem  urban 
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area.  Nor  does  merely  representing  buildings  as  solid  objects  suffice.  A  key  feature  of 
the  urban  environment  is  the  complexity  created  by  the  presence  of  structures  that 
provide  concealment  and  firing  positions.  Structurally  they  must  possess,  multiple  levels, 
allow  for  realistic  structural  damaged,  fortification,  etc.  Thus,  it  is  desirable  to  represent 
a  building  as  a  fully  interactive  three-dimensional  structure  with  openings  that  allow 
entry  and  exit  of  entities  and  projectiles,  occupation  of  and  travel  between  multiple 
levels,  and  realistic  damage  based  on  weapons  effects  (Miller,  2003). 

In  addition  to  the  buildings  and  terrain,  a  JUO  training  environment  must  include 
realistic  simulation  of  non-combatant  entities.  The  requirement  to  minimize  civilian 
casualties  and  the  need  to  identify  combatants  concealing  themselves  within  the 
population  demands  an  accurate  depiction  of  non-combatants.  These  entities  must 
behave  in  a  culturally  consistent  manner  performing  daily  activities  on  a  realistic 
timetable  (Dehncke,  2005).  However,  modeling  crowds  of  people  goes  beyond 
accurately  simulating  a  single  individual’s  behavior  under  a  given  situation.  Accurately 
modeling  a  crowd  requires  allowances  for  different  entities  to  have  different  motivations 
and  behaviors  when  faced  with  the  same  situation.  A  model  that  includes  identically 
behaving  entities  will  not  appear  realistic  even  if  a  single  entity  behaved  in  a  realistic 
manner  (Ulicny,  2001). 

Further  complicating  the  problem  for  military  simulation  is  the  need  to  model 
enemy  forces  in  the  urban  environment.  No  longer  is  it  a  simple  matter  to  simulate  the 
behavior  of  an  enemy  combatant  engaging  in  open  combat  using  conventional  warfare 
tactics.  Although  urban  operations  can  take  the  fonn  of  open  combat  with  a  clear  enemy, 
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the  more  difficult  task  of  defending  against  an  insurgency  or  perfonning  peacekeeping 
operations  is  becoming  more  prevalent  as  was  demonstrated  during  operations  in  Kosovo, 
Mogadishu,  and  now  Iraqi.  The  challenge  in  such  environments  is  to  model  the 
insurgents  as  they  attempt  to  blend  in  while  performing  clandestine  nefarious  activities. 

Synthetic  Environment  and  Semi-Automated  Forces 

In  the  past,  senior  leaders  were  content  to  practice  the  art  of  war  by  participating 
in  what  could  best  be  described  as  glorified  games  of  Risk,  controlling  the  movements  of 
divisions,  corps,  and  even  entire  armies  as  the  battle  progresses.  The  need  for  senior 
leaders  to  train  in  large-scale,  yet  highly  detailed,  battle  spaces  has  become  a  critical 
issue.  Senior  leaders  and  strategists  must  have  the  ability  to  train  for  the  more  complex 
and  intricate  environments  facing  them  today  and  in  the  future.  The  explosion  of  ISR 
assets  and  the  transfonnation  to  network  centric  warfare  has  provided  unique 
opportunities  but  also  unique  challenges.  The  ability  to  obtain  vast  quantities  of  data  is 
an  incredible  achievement,  however,  data  is  not  information,  and  information  is  not 
understanding. 

The  ability  to  generate  large-scale  high-resolution  synthetic  environments  is 
relatively  new.  Although  combat  modeling  state-of-the-art  has  been  marching  at  a  steady 
pace  towards  such  a  goal,  it  is  only  recently  that  all  the  pieces  required  have  fallen  into 
place.  Much  of  the  credit  can  be  assigned  to  the  huge  advances  in  computing  power  that 
have  occurred  in  recent  years.  However,  just  as  important  as  the  advances  in  technology 
have  been,  so  has  the  evolution  of  the  need  for  such  environments. 
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URBAN  RESOLVE 


In  response  to  the  DoD’s  need  for  realistic  urban  training  and  improved  tactics 
and  systems,  the  U.S.  Joint  Forces  Command’s  (JFCOM)  Joint  Experimentation 
directorate  (J9)  sponsored  the  experiment  URBAN  RESOLVE  in  2004.  URBAN 
RESOLVE  is  a  three-phase  effort,  extending  to  2007  and  beyond,  with  the  goal  to, 
guide  the  development  of  critical  warfighting  capabilities  for  the  future  joint  force 
commander,  with  a  particular  focus  on  those  needed  for  effective  urban  operations,” 
(Urban,  2006).  The  URBAN  RESOLVE  experiment  takes  place  within  a  massive 
synthetic  urban  environment  that  consists  of  1.8+  million  building  structures.  Of  those 
structures,  65,000  of  them  allow  for  increased  interaction  such  that  entities  can  enter  the 
buildings,  fight  inside,  and  see  the  street  from  within  the  building.  A  key  aspect  of  the 
synthetic  environment  in  URBAN  RESOLVE  is  the  simulation  of  a  dense  urban 
population,  called  cultural  features,  that  included  over  124,000  entities  with 
approximately  35,000  of  them  active  and  behaving  in  a  culturally  accurate  manner. 
Testing  has  shown  that  over  1  million  entities  are  possible.  Green  Force  participants 
controlled  the  cultural-features  (Wielhouwer,  2005:12-13  and  Rafuse,  2006). 

The  first  phase  of  URBAN  RESOLVE,  URBAN  RESOLVE  2015,  occurred 
between  June  and  October  2004.  URBAN  RESOLVE  2015’s  goal  was  to  assess  the 
ability  of  the  Joint  Forces  Commander  to  attain  situational  awareness  utilizing  a  myriad 
of  current  and  future  ISR  assets.  The  Blue  Forces’  mission  was  to  employ  ISR  assets  to 
detect  and  predict  the  movements  and  presence  of  Red  Force  entities  as  they  moved  about 
within  the  dense  cultural  features  of  the  synthetic  environment,  and  to  stealthily  deploy 
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and  conceal  their  ISR  assets.  Meanwhile  the  Red  Forces’  mission  was  to  challenge, 
adapt,  and  react  to  the  Blue  Force  in  a  manner  consistent  with  an  aggressive  and  adaptive 
enemy  (Wielhouwer,  2005:10-1 1).  The  results  from  Phase  1  were  impressive.  The  Blue 
Force  detected  80%  of  the  Red  Forces,  identified  50%-60%  of  the  Red  Forces’  current 
activities,  and  anticipated  60%-70%  of  their  future  actions  (Wielhouwer,  2005:14). 

The  backbone  technologies  that  made  URBAN  RESOLVE  possible  were  the 
JSAF  synthetic  environment,  the  use  of  distributed  Scalable  Parallel  Processors  (SPP) 
clusters,  the  Defense  Research  Engineering  Network  (DREN),  and  the  Command  and 
Control  (C2)  workstations  used  for  HITL  warfighter  involvement  (Dehncke,  2005).  The 
JSAF  environment  was  responsible  for  providing  the  synthetic  environment  in  which  the 
exercise  occurred  as  well  as  the  entity  level  Blue  and  Red  Forces,  and  the  Green  team’s 
cultural  features.  Modeling  of  the  sensor  data  was  perfonned  by  the  Simulation  of 
Location  and  Attack  of  Mobile  Enemy  Missiles  (SLAMEM)  federate.  Modeling  of 
HUMINT  data  was  provided  by  the  SOAR-SOF  federate.  The  SPP  clusters  provided  the 
raw  processing  power  required  for  JSAF  to  generate  the  environment  and  entities.  The 
SPP  clusters  are  located  at  the  Maui  High  Performance  Computer  Center  (MHPCC)  in 
Maui,  HI  and  the  Aeronautical  Systems  Center  Major  Shared  Resource  Center  (ASC- 
MSRC)  at  Wright  Patterson  Air  Force  Base,  OH.  Additional  computing  resources  for 
Blue,  Red,  and  Green  Force  operations,  simulation  control,  and  the  analysis  team  were 
located  at  Ft  Belvoir,  VA,  San  Diego  CA,  and  Suffolk,  VA.  Linking  all  these  distant 
resources  together  in  real-time  was  the  DREN  as  depicted  in  Figure  2  (Dehncke,  2005). 
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Figure  2.  Connectivity  between  distributed  participants  (Dehncke,  2005) 


The  C2  systems  for  warfighter  interaction  consist  of  a  JSAF  display  responsible 
for  providing  the  shared  tactical  picture  as  perceived  by  the  participants,  a  JSAF  control 
terminal  for  sensor  controls,  data  display  assessment,  track  management,  and  a  terminal 
for  collaborative  tools  such  a  text,  chat,  voice  and  graphics  (Figure  3). 
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Figure  3.  Operator’s  C2  Workstation  (Dehncke,  2005) 


The  process  of  generating  a  track  in  URBAN  RESOLVE  involves  the  generation 
and  transmission  of  “ground  truth”  data  for  all  entities  (Blue,  Red,  and  Green  Forces).  As 
these  entities  move  around  the  synthetic  environment,  they  may  enter  the  sensor  footprint 
of  an  ISR  entity.  At  this  point,  the  simulation  decides  if  the  entity  is  visible  to  the  sensor 
based  on  various  criteria  such  as  line-of-site  (LOS).  If  the  entity  is  detectable  by  the 
sensor,  this  triggers  an  “interaction”  event.  The  interaction  event  is  then  sent  to  a 
confusion  matrix,  in  the  SLAMEM  federation,  that  adds  a  level  of  uncertainty  to  the 
interaction.  The  confusion  matrix  uses  data  related  to  the  sensors  capabilities  against  the 
given  target  entity,  environmental  conditions,  etc.  and  generates  a  track  indicating  the 
possible  identity  of  the  track  and  a  confidence  value.  The  tracks  are  then  displayed  for 
the  warfighters  on  their  consoles.  As  multiple  sensor  hits  accumulate  for  a  given  track,  a 
process  within  SLAMEM  automatically  correlates  tracks  to  increase  the  confidence  level 
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of  the  track’s  label.  An  example  would  be  three  tracks  generated  by  three  different 
sensors  on  a  given  moving  target.  When  considered  separately,  each  of  those  tracks  may 
have  a  low  confidence  that  the  target  in  question  is  a  tank.  However  when  considered 
together,  those  three  tracks  may  indicate,  with  a  high  level  of  confidence,  that  the  track  in 
question  is  a  T-72  tank  (Rafuse,  2006). 

As  the  warfighters  observe  the  multitude  of  tracks  on  their  consoles,  they  can 
make  inferences  as  to  the  intent  of  the  track(s).  These  inferences  are  called  situational 
awareness  objects  (SAO)  that  can  be  considered  notional  “buckets”  that  contain  a  track, 
or  tracks.  An  SAO  indicates  the  operator’s  guess  as  to  the  track’s  intent  and  provides  a 
mechanism  for  the  operator  to  share  this  infonnation  with  other  players.  For  example,  a 
series  of  tracks  labeled  as  vehicles,  containing  armed  civilians,  known  insurgent  leaders, 
and  traveling  together  along  the  same  route,  may  be  labeled  as  an  enemy  convoy  and 
placed  in  an  SAO.  Analysis  of  these  SAOs  is  what  allowed  the  exercise  evaluators  to 
detennine  if  the  Blue  Force  was  able  to  accurately  determine  and  predict  the  activities  of 
the  Red  Force  (Rafuse,  2006). 

What  should  become  apparent  from  the  discussion  thus  far  are  the  vast  quantities 
of  data  such  an  exercise  generates.  The  dataset  for  a  single  twenty-four  hour  simulation 
trial,  excluding  all  cultural-feature  data,  was  approximately  ninety  to  one  hundred  twenty 
gigabytes  containing  forty  to  fifty  million  rows  of  data.  The  entire  URBAN  RESOLVE 
2015  Phase  1  exercise  consisted  of  seven  such  trials.  The  complete  dataset,  including  all 
cultural-features,  consumed  over  3.5  terabytes  of  disk  space.  This  presents  a  significant 
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obstacle  for  data  analysis  and  presents  opportunities  for  new  techniques  in  analysis  and 
data  mining. 

Collecting  such  a  vast  amount  of  data  presented  certain  obstacles  as  well.  Early 
data  collection  schemas  attempted  to  push  all  logged  data  to  a  central  repository. 
However,  as  the  exercise  environment  grew,  the  data  transfer  requirement  overloaded  the 
physical  network  and  resulted  in  data  loss.  In  addition,  it  was  not  possible  to  log  some 
simulation  event  data  because  of  the  way  cluster  computers  operate.  To  address  these 
limitations  URBAN  RESOLVE  Phase  I  utilized  interceptor/logger  applications  operating 
at  each  computing  cluster.  An  aggregator  application  was  used  to  request  data  of  interest 
from  the  individual  computing  clusters  (Graebener,  2004:2-3).  The  result  was  a  data 
collection  schema  that  allowed  robust  collection  without  saturating  the  network  and 
afforded  the  analysts  near-real-time  access  to,  and  analysis  of,  the  data.  The  backbone  of 
the  database  was  the  open  source  database  engine  MySQL.  The  open  source  nature  of 
MySQL  allowed  the  analysts  to  modify  the  actual  application  to  suit  their  needs. 

The  initial  intent  of  URBAN  RESOLVE  was  as  a  tool  to  evaluate  the  ability  of 
current  and  future  ISR  assets  to  aid  the  warfighter  in  gaining  situational  awareness  in  a 
modem  urban  setting.  The  success  of  Phase  1  showed  senior  leaders  the  potential  of 
applying  URBAN  RESOLVE’S  experimental  approach  to  problems  faced  in  present  day 
urban  operations.  The  result  was  an  expansion  of  URBAN  RESOLVE  to  establish  a 
baseline  of  current  urban  warfare  capabilities,  and  address  current  challenges  faced  by 
our  forces.  The  two  new  directives  included  adapting  the  modeling  and  simulation 
framework  to  support  its  use  as  a  mission  planning  and  rehearsal  tool  for  deploying 
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forces,  and  supporting  development  of  more  effective  responses  to  non-traditional  mortar 
attacks  in  Iraq  (Wielhouwer,  2005:1 1-12). 

Summary 

This  literature  review  investigated  the  current  scope  of  modeling  and  simulation 
in  the  DOD  and  the  history  of  combat  modeling.  The  review  then  continued  with  a 
discussion  of  urban  combat  and  the  DoD’s  need  to  better  train  and  equip  our  forces  to 
operate  in  the  urban  environment.  This  led  to  an  examination  of  computer  generated 
synthetic  environments,  semi-automated  forces,  and  ultimately  the  URBAN  RESOLVE 
exercise.  Finally,  this  literature  review  showed  that  data  collection,  reduction  and 
analysis  is  a  significant  challenge  to  such  a  large-scale  event. 
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III.  Methodology 


Chapter  Overview 

This  purpose  of  this  chapter  is  to  provide  an  overview  of  the  methodology 
employed  for  the  URBAN  RESOLVE  exercise  and  the  methods  used  in  this  research 
effort  to  analyze  the  data  generated  from  the  exercise.  It  begins  with  a  discussion  of 
URBAN  RESOLVE’S  experimental  architecture,  experimental  design,  data  collection 
methods,  and  data  analysis.  This  chapter  concludes  with  a  discussion  of  the  limitations  of 
the  available  data,  a  nonparametric  method  for  generating  confidence  intervals  for  the 
median  of  single-population  data,  and  finally  a  description  of  the  analysis  performed. 

Experiment  Architecture 

As  previously  discussed,  URBAN  RESOLVE  Phase  I  was  an  exercise 
intended  to  investigate  the  utility  of  new  and  emerging  ISR  technologies,  and  the  TTPs 
used  to  employ  them,  in  a  large-scale  HITL  synthetic  urban  environment.  The  URBAN 
RESOLVE  exercise  took  place  in  the  simulated  Indonesian  capital  city  of  Jakarta.  The 
Red  Forces  withdrew  into  the  city  to  mount  a  final  defense  and  possess  medium-range 
ballistic  missiles  (MRBM).  The  Blue  Force  Joint  Intelligence  and  Fusion  Cell  (Blue 
Cell)  controlled  an  array  of  sensor  platforms  that  included  unmanned  aerial  vehicles 
(UAV),  organic  aerial  vehicles  (OAV),  and  unmanned  ground  sensors  (UGS).  The  Blue 
Cell  controlled  these  platfonns  to  gain  situational  awareness  of  the  enemy’s  activities. 

The  UAVs  operated  at  medium/high  altitude,  provided  a  “persistent”  stare  into  the  area  of 
interest  (AOI),  and  were  responsible  for  initiating  and  maintaining  tracks  of  Red  entities. 
The  OAVs  operated  a  low  altitude  were  ale  to  flow  down  into  the  AOI  and  provided  a 
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closer  look  to  locate,  track,  and  identify  Red  entities.  The  OAVs  were  also  capable  of 
placing  radio  frequency  tags  onto  vehicles  and  personnel.  The  UGSs  assisted  the  Blue 
cell  in  locating,  tracking,  and  identifying  Red  entities,  and  in  some  cases  tagging  entities 
(Burke,  2005:9).  A  broad  range  of  sensors  were  available,  and  included  acoustic/seismic, 
magnetometer,  Ku-band  moving  target  indicator  (MTI),  Ku-band  synthetic  aperture  radar 
(SAR),  foliage  penetrating  SAR,  LADAR,  Laser  Profilers,  Electro-optical/Infra-red 
(EO/IR),  and  X-ray  sensors.  In  addition,  special  operations  forces  (SOF)  and  HUMINT 
were  available  (Burke,  2005:24-30).  Tracks  within  URBAN  RESOLVE  were  generated 
using  the  following  process: 

•  A  sensor  foot  print  was  projected  onto  the  AOI 

•  If  an  entity  fell  within  that  foot  print  it  was  tested  for  LOS,  velocity,  and 
concealment 

•  If  the  entity  was  deemed  “visible”  to  the  sensor  the  following  occurred: 

o  A  random  draw  was  made  and  compared  against  the  probability  of 
detection  for  the  sensor-target  pairing  to  detennine  if  the  sensor 
actually  detects  a  target 

o  If  a  detection  occurred  the  entity  was  processed  through  a 
confusion  matrix  to  detennine  how  the  target  is  classified 

o  The  track  was  then  either  passed  to  the  Blue  Cell,  updated  if  the 
track  already  existed,  or  used  to  cue  another  sensor-platfonn  pair 

Target  conelation  was  modeled  perfectly  such  that  two  or  more  sensors  that 
detected  the  same  target  resulted  in  a  single  track.  Similarly,  two  targets  were  never 
classified  as  the  same  target  if  they  were  in  close  proximity  (Burke,  2005:3 1-32). 
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Experiment  Design 

URBAN  RESOLVE  Phase  I  exercise  encompassed  seven  different  scenarios. 

The  exercise  variables  were  Blue  Force  platforms  inventory,  target  signature,  Red  Force 
activity  level,  and  availability  of  Tags.  The  complete  exercise  design  is  shown  in  Table 
1 .  The  parameter  “reduced  Blue  platform  inventory”  meant  the  removal  of  all  UAVs 
with  hyperspectral  and  EO/IR  sensors  from  the  simulation.  The  remaining  platforms 
were  the  UAVs  with  SAR  sensors.  The  intent  of  the  reduction  was  to  simulate  a  low 
cloud  ceiling.  The  parameter  for  “Red  Force  signature  reduction”  simulated  attempts  by 
the  Red  Force  to  reduce  or  alter  their  signature  return.  The  reduced  signature  effect  was 
modeled  in  the  simulation  by  modifying  the  detection  and  identification  capabilities  of 
Blue  Force  sensors  relative  to  those  targets.  Red  Force  activity  level  (active/inactive) 
indicates  the  proportion  of  the  force  that  began  the  event  outside  the  AOI.  An  active  Red 
Force  started  the  event  with  70%  of  its  forces  outside  the  AOI  while  an  inactive  force 
started  with  70%  of  forces  already  inside  the  AOI.  The  presence  of  Tags  in  the  exercise 
was  varied  by  allowing  or  disallowing  their  use  (Burke,  2005:ES7-ES9).  Tags  are  small 
radio  frequency  ID  (RFID)  tags  that  are  attached  to  enemy  personnel  and  equipment  by 
either  OAVs  or  UGSs.  Blue  Force  sensors  can  track  the  Tags  and  provide  an  effective 
method  for  maintaining  situational  awareness  of  a  track’s  position  and  identity  even  if  the 
track  is  lost  for  extended  periods  within  the  urban  clutter. 

An  inspection  of  Table  1  reveals  a  flaw  in  the  experimental  design.  In  each  trial, 
more  than  a  single  parameter  is  changed.  Designing  trials  with  a  multiple  parameter 
changes  makes  it  difficult  to  correlate  observed  changes  in  the  data  to  changes  in  a 
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particular  parameter.  A  compounding  factor  to  the  poor  experimental  design  is  the  lack 
of  multiple  replicate  runs  for  each  trial.  This  is  expected  from  an  exercise  of  this  type 
due  to  the  large  amount  of  time  and  resources  required  to  perfonn  a  single  replication. 
With  only  single  replications,  that  are  not  independent,  classical  statistical  comparative 
analysis  is  not  possible. 


Table  1.  Exercise  Design  (Burke,  2005: 16-17) 


Trial  1 

•  Full  blue  platform  inventory 

•  Full  target  signature 

•  Active  Red 

•  48-hour  trial 

•  Tags 

Trial  2A 

Trial  2B 

• 

Full  Blue  platform  inventory 

•  Reduced  Blue  platform  inventory 

• 

Full  targets  signature 

•  Full  target  signature 

• 

Inactive  Red 

•  Inactive  Red 

• 

Tags 

•  Tags 

• 

24-hour  trial 

•  18-hour  trial 

Trial  3A 

Trial  3B 

• 

Full  Blue  platform  inventory 

•  Reduced  Blue  platform  inventory 

• 

Reduced  target  signature 

•  Full  target  signature 

• 

Active  Red 

•  Active  Red 

• 

Tags 

•  Tags 

• 

24-hour  trial 

•  24-hour  trial 

Trial  4 A 

Trial  4B 

• 

Full  Blue  platform  inventory 

•  Reduced  Blue  platform  inventory 

• 

Full  target  signature 

•  Reduced  target  signature 

• 

Active  Red 

•  Active  Red 

• 

No  Tags 

•  No  Tags 

• 

24-hour  trial 

•  24-hour  trial 

Data  Collection  and  Reduction 

Data  collection  in  URBAN  RESOLVE  was  a  monumental  task.  In  any 
given  event,  the  constellation  of  Blue  Force  platforms  numbered  between  115  and  330, 
many  of  which  had  multiple  sensors.  The  constellation  was  reporting  detections  on  over 
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1 100  Red  Force  entities  dispersed  within  a  population  of  over  100,000  cultural  features 
(Burke,  2005:77-80).  Each  event  run  generated  approximately  40-50  million  rows  of 
data  and  consumed  90-120  gigabytes  of  physical  storage  space  (Rafuse,  2006).  Table  2 
shows  an  example  of  one  of  the  many  tables  available.  This  particular  table  shows  the 
data  generated  from  a  single  contact  report  when  a  sensor  detects  an  entity. 
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Table  2.  Raw  MySQL  data  table  from  URBAN  RESOLVE  Phase  I  (2005) 


Field 

DataType 

Description 

uniquelndex 

STRING 

Unique  ID  for  record  within  individual  logger  database 

node 

STRING 

Node  hostname 

spaceName 

INTEGER 

RTI  Namespace  Name 

timestamp 

FLOAT 

Local  machine  timestamp  for  record  (when  written) 

platform  id  ID 

STRING 

Identifies  the  platform  this  sensor  is  mounted  on. 

sensor  id 

INTEGER 

Sensor  ID  on  this  platform 

sensor_plan  mode 

INTEGER 

Internal  ID  of  sensor/mode/exploitation  combination 

batch  flag 

INTEGER 

Internal  use  (beg/middle/end  of  collections) 

FOPEN  depth 

FLOAT 

Two-way  detection  threshold  (meters) 

owning_force 

STRING 

Force  of  entity  object  (Friendly/Opposing/Neutral) 

platform  type  EntityKind 

INTEGER 

Enumeration  value  for  entity  object  -  Kind 

platform  type  Domain 

INTEGER 

Enumeration  value  for  entity  object  -  Domain 

platformtypeCountryCode 

INTEGER 

Enumeration  value  for  entity  object  -  Country 

platform  type  Category 

INTEGER 

Enumeration  value  for  entity  object  -  Category 

platform_type_Subcategory 

INTEGER 

Enumeration  value  for  entity  object  -  SubCategory 

platform_type_Specific 

INTEGER 

Enumeration  value  for  entity  object  -  Specific 

platform  type  Extra 

INTEGER 

Enumeration  value  for  entity  object  -  Extra 

sensor  location  X 

DOUBLE 

GCS  Location  of  entity  object  -  X  Value 

sensor  location  Y 

DOUBLE 

GCS  Location  of  entity  object  -  Y  Value 

sensor  location  Z 

DOUBLE 

GCS  Location  of  munition  object  -  Z  Value 

platform  velocity  Xvelocity 

FLOAT 

Movement  velocity  of  entity  object  -  X  Value 

platform  velocity  Yvelocity 

FLOAT 

Movement  velocity  of  entity  object  -  Y  Value 

platform  velocity  Zvelocity 

FLOAT 

Movement  velocity  of  entity  object  -  Z  Value 

sensor  mode 

STRING 

Mode  of  operation  for  sensor  (MTI,  STRIP  EO,  etc.) 

comment 

STRING 

The  comment  to  attach. 

entity  id  ID 

INTEGER 

Ground  truth  id  of  the  entity  that  got  painted. 

location  X 

DOUBLE 

Ground  truth  GCS  Location  of  entity  object  -  X  Value 

location  Y 

DOUBLE 

Ground  truth  GCS  Location  of  entity  object  -  Y  Value 

location  Z 

DOUBLE 

Ground  truth  GCS  Location  of  entity  object  -  Z  Value 

velocityXVelocity 

FLOAT 

Movement  velocity  of  entity  object  -  X  Value 

velocity  YVelocity 

FLOAT 

Movement  velocity  of  entity  object  -  Y  Value 

velocity  ZVelocity 

FLOAT 

Movement  velocity  of  entity  object  -  Z  Value 

appearance 

INTEGER 

Appearance  bits  of  the  painted  vehicle. 

building_attributes 

INTEGER 

Attribute  bits  of  the  painted  building. 

detectionstatus 

STRING 

Enum'd  result  of  the  filters,  indicating  if  the  vehicle  id  detectable  or  not. 

detected  type  EntityKind 

INTEGER 

Enumeration  value  for  the  true  guise  -  Kind 

detected  type  Domain 

INTEGER 

Enumeration  value  for  the  true  guise  -  Domain 

detected  type  CountryCode 

INTEGER 

Enumeration  value  for  the  true  guise  -  Country 

detectedtypeCategory 

INTEGER 

Enumeration  value  for  the  true  guise  -  Category 

detected_type_Subcategory 

INTEGER 

Enumeration  value  for  the  true  guise  -  SubCategory 

detectedtypeSpecific 

INTEGER 

Enumeration  value  for  the  true  guise  -  Specific 

detected  type  Extra 

INTEGER 

Enumeration  value  for  the  true  guise  -  Extra 
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Obtaining  the  URBAN  RESOLVE  data  presented  some  significant  challenges  for 
this  research  effort.  Due  to  the  size  of  the  dataset,  we  were  not  able  to  request  the  entire 
dataset  on  a  portable  media  format  such  as  a  DVD  or  CD.  Initially  we  tried  to  work  with 
a  very  small  subset  of  the  data  received  on  DVD.  However,  due  to  database  software 
differences,  a  significant  amount  of  time  was  expended  to  make  the  data  usable  with  only 
partial  success.  Even  then,  we  quickly  realized  that  the  subset  of  data  was  not  sufficient 
to  perform  any  meaningful  analysis.  The  dataset  contained  only  a  four-hour  block  of 
time  from  a  single  trial.  Eventually  we  were  able  to  gain  direct  access  to  the  entire 
dataset  via  the  J9  servers  housing  the  data. 

Post  analysis  data  reduction  of  such  large  quantities  of  data  presented  similar 
challenges.  As  with  data  collection,  early  data  analysis  techniques  proved  problematic  as 
the  datasets  grew  in  size.  However,  the  URBAN  RESOLVE  analysts  were  able  to  use 
another  open  source  tool  called  PHP  (recursive  acronym  for  PHP:  Hypertext  Protocol)  to 
extract  the  specific  data  required  to  answer  the  exercise’s  measures  of  performance 
(MOP)  (PHP,  2006).  Table  3  shows  a  small  portion  of  the  data  that  was  aggregated  from 
the  raw  dataset.  The  entire  table  contains  28,815  rows.  Of  particular  interest  to  this 
research  are  the  “initial  detection  time”,  “initial  identification  time”,  “target  type”,  and 
“track  originating  platform.”  The  exercise  planners  wished  to  answer  three  top-level 
questions: 

1)  To  what  extent  can  situational  awareness  be  developed  during  JUO? 

2)  How  effectively  are  sensor  data  used  to  detect  and  correlate  targets? 

3)  What  conditions  affect  Blue  capabilities? 


26 


Table  3.  Track  Data  table  URBAN  RESOLVE  Phase  I  Trial  1  (2005) 


This  research  most  directly  relates  to  question  three.  Each  question  had  a  set  of 
MOPs  associated  with  it  that  are  meant  to  help  answer  these  top-level  questions. 
Similarly,  each  MOP  had  approximately  twelve  measures  that  were  used  to  evaluate  the 
MOP.  Table  4  provides  an  example  of  the  types  of  MOPs  and  measures  used  in  URBAN 
RESOLVE.  Appendix  A  contains  the  entire  set  of  MOPs  and  measures. 


Table  4.  Example  of  URBAN  RESOLVE  MOP  and  Associated  Measures 


Ql.  To  what  extent  can  SA/SU  be  developed  in  JUO 

Ml. 2  Number  of  Critical  Weapon  Nodes  Identified 

a 

Terminal  points  of  Red  vehicle  paths  (see  Ml.l) 

b 

Terminal  points  of  Red  DI  paths  (see  Ml.l) 

c 

Acquisitions  of  Red  weapon  systems  (characteristics) 

d 

Acquisitions  of  Red  weapon  systems  (location) 

e 

Acquisitions  of  Red  WMD  systems  (characteristics) 

f 

Acquisitions  of  Red  WMD  systems  (location) 

Data  Analysis 

The  analysis  performed  in  this  research  will  focus  on  two  areas:  the  time  to 
identify  (TTI)  a  target  once  it  has  been  detected  and  the  rate  at  which  identifications  are 
generated  versus  time.  Each  area  will  be  evaluated  for  patterns  or  trends  between  trials, 
target  groups,  and  detecting  sensor  type.  As  mentioned  previously  the  level  of  effort  and 
resources  required  to  perfonn  a  given  URBAN  RESOLVE  event  meant  that  only  a  single 
replication  was  performed  for  each  exercise  design  trial.  Analysis  of  single  replication 
data  presents  several  difficulties  for  an  analyst.  The  assumption  that  the  trials  are 
independent  or  identical  cannot  be  made  because  exercise  participants  remained  the  same 
for  all  trials.  Ordinarily  this  may  not  present  a  problem,  however  a  definite  learning 
curve  was  observed  in  relation  to  the  proficiency  of  the  players.  As  the  exercise 
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progressed,  the  Blue  and  Red  force  operators  became  more  efficient  at  using  the 
simulation  hardware  as  well  as  more  adept  at  playing  the  game.  The  result  was  the 
exclusion  of  Trial  1  from  the  original  analysis  and  the  loss  of  the  intended  “base  case.” 

In  addition  to  the  unintentional  “learning”  that  was  observed,  exercise  participants  on 
both  sides  intentionally  varied  their  tactics.  Specifically  the  Blue  force  was  responsible 
for  positioning  their  available  ISR  assets  before  the  start  of  each  trial.  The  Red  force  was 
also  encouraged  to  vary  their  tactics  to  keep  the  element  of  surprise  (Burke,  2005:66). 
Assuming  that  the  distributions  of  the  TTIs  for  each  design  case  are  the  same  may  or  may 
not  be  reasonable.  However,  given  the  similar  process  that  generated  them,  we  assume 
they  are  close  enough  for  our  purposes.  The  distributions  are  not  nonnal  and  the  central 
limit  theorem  cannot  be  used  because  we  lack  identical  and  independent  replications  for 
each  design  trial  (Wackerly,  2002:346-348).  The  distribution  variances  cannot  be 
assumed  the  same  either. 

Although  many  assumptions  about  the  data  cannot  be  made,  nonparametric 
statistics  may  still  be  used  to  perfonn  statistical  analysis  (Wackerly,  2002:708-709). 

Many  nonparametric  statistics  only  require  that  the  data  be  continuous  and  random 
(Neter,  1988:481).  Unfortunately,  in  order  to  perform  any  form  of  hypothesis  testing  still 
requires  independence.  A  nonparametric  confidence  interval  for  the  population  median 
can  be  determined  for  a  large-sample  (n  >15)  single-population  using  the  following 
method;  all  data  in  this  analysis  contained  at  least  15  data-points.  For  a  desired  1  -  a 
confidence  coefficient  for  confidence  interval  L,-<r\<  Ur  such  that  L,  and  Ur  are  the  rth 
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smallest  and  largest  samples  respectively.  The  variable  r  is  the  largest  integer  that  does 
not  exceed  the  value  of  equation  (1)  (Neter,  1988:485). 

r  =  0.5 [//  +  1  -  z(l  -  a/2)Vn]  (1) 

A  large  majority  of  the  TTI  values  are  zero  because  many  target  identifications 
are  generated  by  the  same  sensor  that  initially  detected  them.  Therefore,  it  is  more 
appropriate  to  use  the  median  TTI  rather  than  the  mean  TTI.  The  median  is  the  middle  of 
a  distribution  such  that  half  of  all  values  are  above  and  below  it.  This  makes  the  median 
much  less  susceptible  to  extreme  values  than  the  mean  and  thus  more  appropriate  for  this 
study. 

Summary 

This  chapter  introduced  the  general  experimental  architecture  and  design  of 
URBAN  RESOLVE  Phase  I  and  pointed  out  the  limitations  of  an  experimental  design 
where  more  than  a  single  parameter  is  varied  in  each  trial.  Next,  it  discussed  data 
collection,  data  reduction,  and  the  difficulties  of  working  with  such  a  large  and  expanding 
dataset.  Next,  this  chapter  discussed  the  limitations  of  working  with  single-replication 
datasets  that  preclude  typical  parametric  statistical  methods  of  analysis.  Finally,  this 
chapter  discussed  a  nonparametric  method  to  generate  confidence  intervals  for  the 
median  value  of  that  data. 
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IV.  Analysis  and  Results 


Chapter  Overview 

This  chapter  details  the  data  analysis  perfonned  on  the  data  from  URBAN 
RESOLVE  Phase  I  for  this  research.  Where  applicable  the  analysis  involves  not  only  the 
complete  dataset  for  a  given  trial,  but  also  decomposition  by  target-type  and  the  track- 
originating  sensor. 

Data  Analysis 

URBAN  RESOLVE  Phase  I  consisted  of  seven  different  trials  of  which  six 
datasets  were  obtained  (Trial  3b  was  missing).  Within  each  trial  folder  there  was  a  file 
containing  the  complete  list  of  Red  Force  tracks  generated  during  a  run.  Each  record 
included  the  platform,  sensor,  and  time  for  track  detection  and,  when  available,  track 
identification.  Track  identification  data  was  not  available  if  the  track  was  not  identified. 
This  analysis  focuses  its  efforts  only  on  tracks  that  were  identified.  The  data  was 
manipulated  using  Microsoft  Excel  to  sort  the  tracks  by  identifying  time  and  the  TTI  was 
calculated  from  the  difference  between  the  track  identifying  and  originating  times.  The 
tracks  were  sorted  again  based  on  TTI  and  the  process  was  repeated  for  the  remaining 
trial  datasets. 

Initially  we  considered  the  mean  TTI  for  all  tracks.  However,  the  data  included  a 
significant  number  of  zero  data  points  which  skewed  the  data  (i.e.  eighty-nine  percent  of 
all  tracks  in  Trial  4b  possessed  a  TTI  of  zero).  A  TTI  =  0  meant  the  track  was  identified 
immediately  by  the  detecting  sensor  and  is  tenned  an  “Instant-ID.”  In  an  effort  to 
minimize  the  effects  of  these  extreme  data  points,  the  median  TTI  was  used.  Use  of  the 
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median  statistic  helps  reduce  the  effect  of  extreme  values.  The  median  is  the  middle  data 
point  such  that  there  are  an  equal  number  of  data  points  above  and  below  the  median 
value.  However,  once  the  zeros  were  removed  the  median  TTIs  all  dropped  to  0.0 
minutes  (Table  5). 


Table  5.  All  Target  Types,  All  IDs 


Trial 

Total  Detections 

Total  IDs 

Median  TTI 

Mean  TTI 

Max  TTI 

Instant-ID 

All  Targets 

1 

28517 

10236 

0.0 

5.9 

8398.0 

54% 

2a 

13847 

3352 

0.0 

5.0 

4997.0 

63% 

2b 

7902 

3130 

0.0 

2.2 

4140.0 

66% 

3a 

8159 

3501 

0.0 

12.9 

1425.0 

56% 

4a 

13385 

3888 

0.0 

2.4 

5595.0 

84% 

4b 

11098 

4369 

0.0 

2.1 

5394.0 

89% 

As  a  next  step  in  the  analysis,  the  tracks  were  broken  down  by  target  in  the  hope 
that  some  infonnation  related  to  the  TTI  for  different  targets  may  be  found  within  and 
between  the  trials.  For  the  purpose  of  this  analysis,  and  to  remain  consistent  with 
previous  analysis,  target  types  were  grouped  into  eight  target  groups  as  shown  in  Table  6. 
The  target  groups  categorize  the  targets  such  that  similar  or  related  targets  can  be  viewed 
together. 


32 


Table  6.  Target  Groups 


Target  Group 

Target  Types 

Artillery 

120mm  Mortar 

105mm  Howitzer 

Air  Defense 

AAA  SA-15 

AD  Command  Vehicle  SA-15  Decoy 

Quad  SA18  ZU-23 

SA-1 1  Snow  Drift  Radar 

Mechanized 

BMP-3 

T-90 

T-90  Decoy 

TEL,  WMD,  MTT 

BTR-80  SS26  TEL 

MAZ6430  Truck  SS26  TEL  Decoy 

SS26  Command  Vehicle  SS26  TEL  Reload  Vehicle 

Leaders 

Leaders 

Armed  Civilians 

Armed  Civilians 

Soldiers 

Soldiers 

Others 

Civilian  UNIMOG  Truck  Small  Car 

Military  UNIMOG  Truck  Motorcyclist 

Large  Truck  SUV 

Medium  Truck  SUV  Command  Vehicle 

URAL  Truck  Technical 

ZIL-157  Cargo  Truck 

Now  that  the  data  was  separated  by  target  group,  two  target  groups  did  possess 
median  TTIs  greater  than  zero.  These  target  groups  were  the  Anned  Civilians  (Table  7) 
and  Mechanized  target  groups  (Table  8).  Within  these  two  target  groups  only  Trials  1, 

2b,  and  Trial  3a,  possess  non-zero  median  TTIs.  The  data  for  the  other  target  groups  may 
be  found  in  Appendix  B. 
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Table  7.  Armed  Civilians,  All  IDs 


Trial 

Total  Detections 

Total  IDs 

Lr 

Median  TTI 

Ur 

Max  TTI 

Instant-ID 

Armed  Civilians 

1 

1729 

197 

9.0 

12.8 

15.7 

317.5 

21% 

2a 

1563 

215 

2.7 

3.4 

4.0 

95.2 

50% 

2b 

414 

42 

0.0 

0.0 

2.9 

157.9 

57% 

3a 

3465 

702 

0.0 

0.0 

0.2 

653.8 

52% 

4a 

4959 

609 

0.0 

0.0 

0.0 

171.8 

75% 

4b 

3689 

624 

0.0 

0.0 

0.0 

259.9 

86% 

Table  8.  Mechanized  Targets,  All  IDs 


Trial 

Total  Detections 

Total  IDs 

Lr 

Median  TTI 

Ur 

Max  TTI 

Instant-ID 

Mechanized 

1 

3148 

1802 

0.0 

0.0 

0.0 

333.8 

69% 

2a 

3618 

462 

0.0 

0.0 

0.0 

1081.6 

73% 

2b 

490 

192 

0.0 

0.0 

1.8 

149.8 

55% 

3a 

279 

141 

1.4 

3.9 

7.7 

303.9 

35% 

4a 

518 

256 

0.0 

0.0 

0.0 

249.6 

77% 

4b 

371 

200 

0.0 

0.0 

0.5 

303.6 

58% 

Although  there  are  three  instances,  where  the  median  TTI  was  greater  than  zero, 
no  confidence  level  can  be  associated  with  those  estimates.  Using  the  single-sample 
confidence  interval  discussed  in  Chapter  3,  95%  confidence  intervals  for  the  median  TTI 
were  calculated  for  the  Mechanized  and  Armed  Civilian  target  groups.  The  results  are 
shown  in  Table  7  and  Table  8.  The  column  labeled  ‘r”  shows  the  values  calculated  using 
equation  (1)  while  the  columns  labeled  L,  and  Ur  show  the  lower  and  upper  bounds  of  the 
median  TTI  at  the  95%  confidence  level.  Here  we  see  that  the  only  trial  with  a  lower 
bound  not  equal  to  zero  is  Trial  3a.  We  can  conclude  with  95%  confidence  that  the 
median  TTI  for  Trial  3a  is  non-zero.  However,  a  comparison  with  the  other  trials  is  not 
truly  possible  because  of  the  lack  of  independence.  Not  surprisingly,  Trial  3  a  possessed 
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the  fewest  identifications  with  TTI  =  0  as  evidenced  by  the  35%  Instant-IDs.  Trial  3a  is 
also  the  trial  with  the  fewest  total  IDs. 

As  discussed  earlier,  the  experimental  design  allowed  variation  of  multiple 
parameters  between  each  trial.  As  such,  it  is  difficult  to  make  any  conclusions  about  the 
cause  of  the  non-zero  median  TTI  for  Trial  3a  relative  to  the  other  trials.  It  is  tempting  to 
look  at  Trial  3a’s  reduced  target  signature  parameter  and  conclude  that  was  the  cause  for 
the  non-zero  TTI  and  even  the  lower  number  of  Total  IDs.  However,  Trial  4b  had  both  a 
reduced  signature  parameter  and  a  reduced  sensor  inventory  parameter,  yet  still  possessed 
a  median  TTI  lower  bound  of  0.0  minutes.  Similarly,  comparisons  between  Trial  3a  and 
4a  are  tempting  but  the  absence  of  Tags  makes  it  difficult,  coupled  with  Trial  4a’s  0.0 
minutes  median  TTI  upper  bound. 

Next,  we  focused  on  data  points  where  TTI  was  greater  than  zero.  These  tracks 
required  an  additional  sensor  detection  before  the  target  could  be  identified  with 
confidence.  The  need  for  this  “second-look”  was  the  result  of  conditions  within  the 
synthetic  environment  that  prevented  the  sensor  from  identifying  the  target  with  a  high 
degree  of  confidence.  Another  cause  for  second-look  identifications  were  sensors 
specifically  indented  to  cue  another  sensor  without  producing  a  track  on  the  operator’s 
Shared  Tactical  Picture  display.  Analysis  of  tracks  with  TTI  >  0  was  perfonned  on  the 
Armed  Civilians  and  Mechanized  target  groups.  The  resulting  data  can  be  found  in  Table 
9  and  Table  10  respectively. 
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Table  9.  Armed  Civilians,  TTI  >  0 


Trial 

Total  IDs 

Min  TTI 

Lr 

Median  TTI 

Ur 

Max  TTI 

Armed  Civilians 

1 

155 

0.1 

14.8 

17.4 

22.7 

317.5 

2a 

107 

0.1 

2.1 

2.3 

2.9 

95.2 

2b 

18 

2.1 

4.0 

11.7 

21.7 

157.9 

3a 

339 

0.1 

6.2 

8.6 

13.6 

653.8 

4a 

150 

0.5 

3.4 

3.9 

5.1 

171.8 

4b 

89 

0.1 

1.7 

2.6 

4.4 

259.9 

Table  10.  Mechanized  Targets,  TTI  >  0 


Trial 

Total  IDs 

Min  TTI 

Lr 

Median  TTI 

Ur 

Max  TTI 

Mechanized 

1 

562 

0.1 

12.1 

14.1 

15.5 

333.8 

2a 

124 

0.5 

8.5 

16.7 

31.0 

1081.6 

2b 

87 

0.7 

9.2 

11.7 

14.9 

149.8 

3a 

92 

0.1 

8.0 

10.9 

15.2 

303.9 

4a 

62 

1.6 

17.6 

24.6 

41.2 

249.6 

4b 

88 

0.3 

10.8 

15.4 

19.8 

303.6 

Figure  4  shows  the  median  TTI  along  with  the  95%  upper  and  lower  confidence 
intervals  for  the  Anned  Civilian  target  group.  As  Figure  4  shows,  the  only  trial  that  does 
not  overlap  with  the  majority  of  the  others  is  Trial  1 .  Given  what  we  know  about  the 
learning  curve  experiences  in  the  exercise  this  comes  as  no  surprise.  Although  Trial  2a 
overlaps  with  Trials  2b,  4a,  and  4b,  it  does  have  the  tightest  confidence  interval.  This  is 
an  interesting  result  as  it  is  also  only  the  second  trial  that  occurred  where  there  may  have 
still  be  some  learning  occurring.  Compared  with  the  other  trials,  2a  is  also  the  only  trial 
with  full  inventory  and  signatures  but  an  inactive  force.  Recall  that  the  inactive  force 
parameter  means  70%  of  the  enemy  targets  begin  the  simulation  already  within  the  AOI. 
Further  analysis  would  be  required  to  detennine  any  causal  effect  between  enemy  forces 
that  are  already  inside  the  AOI  vs.  ones  that  could  be  observed  traveling  into  the  AOI. 
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Trial  3a  can  be  seen  outside  the  ranges  of  trials  2a,  4a,  and  4b  and  may  be  an  area  for 
further  study.  Trials  4a  and  4b  do  not  appear  to  be  significantly  different.  This  result  is 
curious  since  when  you  recall  that  Trial  4a  used  the  full  inventory  and  full  signature 
design  variables  while  Trial  4b  used  the  reduced  level  for  both  parameters.  The 
expectation  here  would  be  for  there  to  be  a  noticeable  difference  in  the  ability  of  the  Blue 
force  to  find  and  identify  targets.  The  extremely  wide  range  for  Trial  2b  may  be  due  to 
the  very  low  count  (18)  compared  to  the  other  trials. 


Armed  Civilians  Median  TTI 


Figure  4.  Median  TTI  and  95%  Confidence  Interval  for  Armed  Civilian  Targets 


Figure  5  shows  the  median  TTI  along  with  the  95%  upper  and  lower  confidence 
intervals  for  the  Mechanized  target  group.  The  lower  bound  for  Trial  4a  is  higher  than 
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the  upper  bounds  of  Trials  1,  2b,  and  3a,  but  not  Trials  2a  and  4b.  Unlike  with  the  Armed 
Civilians,  the  confidence  intervals  for  Trials  4a  and  4b  are  close  to  not  overlapping.  As 
mentioned  earlier,  differences  between  these  two  trials  are  expected.  However,  the  result 
is  even  more  curious  than  before  because  the  median  TTI  for  Trial  4a  is  actually  longer 
than  for  Trial  4b.  The  normal  expectation  would  be  for  this  trend  to  be  reversed.  Finally 
comparing  the  Armed  Civilians  and  the  Mechanized  targets,  we  see  that  the  relative  size 
of  the  confidence  intervals  is  inverted  such  that  trials  with  the  larger  intervals  in  Table  9 
possess  the  relatively  smaller  intervals  in  Table  10  and  visa  versa.  Although  there  is  no 
apparent  explanation  for  the  result,  it  begs  the  question  why. 


Mechanized  Median  TTI 


Trial 

1  2a  2b  3a  4a  4b 


Figure  5.  Median  TTI  and  95%  Confidence  Interval  for  Mechanized  Targets 
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Next,  we  looked  at  the  tracks  based  on  the  initial  detecting  platform  (UAV  and 
OAV)  and  the  corresponding  TTI.  The  UGSs  were  not  included  because  they 
represented  a  very  small  proportion  of  the  total  track  detections  (~2%).  Table  1 1  shows 
the  total  number  of  IDs  produced  by  both  platforms,  the  total  number  of  IDs  with  TTI  > 

0,  the  median  TTI  for  all  non-zero  TTIs  and  their  95%  confidence  intervals.  The  first 
item  of  interest  is  the  All  IDs  count.  The  OAVs  produced  significantly  more  IDs  than  the 
UAVs.  This  is  not  a  surprising  result  considering  there  were  approximately  five  times  as 
many  OAVs  in  each  trial  as  there  were  UAVs,  however  the  OAV’s  All  IDs  counts  are  not 
consistently  five  time  larger  than  the  UAVs.  When  looking  only  at  the  IDs  where  TTI  is 
greater  than  zero,  the  OAVs  in  Trial  1  are  the  only  significant  outliers  from  the  rest  of  the 
OAV.  The  fact  that  Trial  1  has  such  a  tight  confidence  interval  and  lower  median  TTI  is 
notable  given  the  learning  that  is  supposed  to  have  occurred  during  Trial  1  as  well  as  the 
large  track  count  which  would  normally  result  in  a  wider  spread  of  values.  Further 
analysis  may  explain  why  detections  generated  by  the  OAVs  in  Trial  1  required  so  little 
time  to  identify  in  comparison  to  the  other  trial.  Unfortunately,  not  much  else  can  be  said 
about  the  data.  The  UAV  TTI  confidence  intervals  all  overlap  with  both  each  other  and 
their  corresponding  OAV  values. 
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Table  11.  TTI  Data  for  UAVs  and  OAVs 


UAV 

OAV 

All  IDs 

Tracks  with  TTI  >  0 

All  IDs 

Tracks  with  TTI  >  0 

Trial 

Count 

Count 

r 

Lr 

Median 

TTI 

Ur 

Count 

Count 

r 

Lr 

Median 

TTI 

Ur 

1 

2715 

1002 

470 

4.4 

5.1 

5.7 

6243 

3040 

1466 

3.5 

3.9 

4.3 

2a 

761 

76 

29 

1.4 

1.7 

2.1 

2410 

1044 

490 

7.8 

8.0 

8.2 

mm 

39 

16 

4 

1.3 

2.2 

15.5 

2563 

651 

300 

6.0 

6.6 

7.9 

3a 

555 

24 

7 

2.3 

4.5 

16.6 

2480 

1155 

544 

4.8 

5.7 

6.2 

4a 

838 

14 

3 

0.5 

3.2 

23.6 

2814 

520 

238 

6.6 

8.1 

9.7 

4b 

897 

50 

18 

2.3 

11.3 

16.0 

3108 

343 

153 

8.0 

9.6 

11.2 

A  final  attempt  to  extract  useful  infonnation  from  the  data  focused  on  the  how 
fast  IDs  were  generated  in  each  trial  for  the  Armed  Civilians  and  Mechanized  Targets;  all 
ISR  platforms  were  included.  Two  charts  were  generated  for  each  target  group,  an 
accumulated  ID  count,  and  a  proportion  of  the  total  ID  count.  Figure  6  and  Figure  7 
show  the  data  for  the  Armed  Civilians  and  Figure  8  and  Figure  9  show  the  data  for  the 
Mechanized  targets.  Note  that  in  Figure  8  the  chart  stops  at  24  hours  in  order  to  better 
illustrate  the  differences  between  the  trials.  Figure  6  shows  a  few  interesting  features. 
Trial  3a  accumulated  identified  targets  significantly  faster  than  all  other  trials.  Again, 
because  of  the  experimental  design  it  is  difficult  to  detennine  an  exact  cause  with  another 
trial  to  compare  to  with  only  single  parameters  being  changed.  However,  it  is  interesting 
to  note  that  Trial  3  a  does  have  the  reduced  signature  parameter  yet  still  generated  more 
targets,  faster  than  all  other  trials.  Similarly,  Trials  1  and  2b  showed  that  significantly 
fewer  tracks  were  accumulated  at  a  slower  rate.  Although  difficult  to  pinpoint  the  cause, 
Trial  1  makes  sense  due  to  learning,  and  Trial  2b  because  of  the  reduced  sensor 
inventory,  inactive  force,  and  only  18  hours  worth  of  data.  Even  if  Trial  2b  had 
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continued  for  another  6  hours,  the  slope  of  the  curve  indicates  it  is  unlikely  to  have 
caught  up  to  3a,  4a,  or  4b.  A  very  interesting  result  is  the  apparent  similarity  between 
Trials  4a  and  4b.  The  two  parameters  that  are  varied  between  these  trials  (inventory  and 
signature)  lead  to  the  natural  assumption  that  Trial  4b  should  accumulate  fewer  tracks  at  a 
slower  rate.  Further  analysis  may  be  warranted  to  determine  why  this  is  occurring. 


Accumulate  IDs  vs  Time 
Armed  Civilians  -  48  Hours 


-o-  Trial  1 

-CD- Trial  2a 

Trial  2b 

-x-  Trial  3a 

Trial  4b 

-o-  Trial  4b  jj 

Figure  6.  Accumulated  IDs  vs.  Time  for  Armed  Civilians 


The  proportional  accumulation  of  Anned  Civilians  shown  in  Figure  7  shows  an 
interesting  trend  for  Trial  1  where  it  appears  that  the  IDs  accumulate  very  slowly  at  first 
then  accelerates  at  an  increasing  rate  after  the  first  24-hours.  Note  that  because  of  the  48- 
hour  run  time  of  Trial  1  it  has  been  displayed  in  two  24-hour  segments.  The  first  segment 
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clearly  shows  a  slower  accumulation  of  identifications  with  a  sudden  increase  in  the  23rd 
hour.  The  second  segment  of  Trial  1  is  on  par  with  the  other  trials.  This  behavior  is 
likely  indicative  of  the  learning  discussed  in  the  original  URBAN  RESOLVE  analysis. 
All  of  the  other  trials  appear  to  accumulate  their  IDs  at  a  similar  and  linear  rate. 
Additional  replications  would  be  required  before  additional  insight  could  be  gained 
concerning  the  shape  of  the  curves.  Trial  2b  was  removed  because  of  its  18-hour 
timeframe. 


Figure  7.  Proportion  of  Total  Accumulated  IDs  vs.  Time  for  Armed  Civilians 


Figure  8  shows  a  rather  different  story  from  Figure  6.  Here  Trial  1  accumulates 
many  more  Mechanized  target  IDs  at  a  much  faster  rate  and  Trial  3a  accumulates  the 
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fewest  IDs  at  the  slowest  rate.  Trials  4a  and  4b  now  show  a  slight  difference  in  both 
accumulation  total  and  rate;  however,  it  is  only  a  slight  difference  especially  considering 
the  inventory  and  signature  difference  between  them  and  compared  to  Trials  1  and  2a. 
Trial  2b  managed  to  accumulate  IDs  at  a  comparable  rate  to  Trials  4a  and  4b,  and  the  data 
at  least  suggests  that  it  would  have  accumulated  a  similar  number  of  IDs  had  it  continued 
for  the  full  24  hours.  Finally,  Trial  2b  compares  more  similarly  with  the  other  trials  and 
shows  a  marked  difference  compared  to  the  Armed  Civilians  where  it  accumulated 
identifications  on  par  with  Trial  1  and  much  slower  than  the  other  trials. 


Accumulated  IDs  vs  Time 
Mechanized  Targets  -  24  hours 


Time 


<>  Trial  1 

-□-Trial  2a 

-e^-  Trial  2b 

-x-  Trial  3a 

— i—  Trial  4a 

-o- Trial  4b 

_ 

Figure  8.  Accumulated  IDs  vs.  Time  for  Mechanized  Targets 
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Finally,  Figure  9  shows  the  proportional  accumulation  of  Mechanized  IDs. 
Again,  the  curve  for  Trial  2b  has  been  removed  because  of  its  18-hour  timeframe.  As 
before,  Trial  1  has  been  split  into  two  24-hour  sections.  The  fist  24  hours  shows  a 
slightly  slower  accumulation  compared  to  the  other  trials  but  not  nearly  to  the  extent  as 
with  the  Anned  Civilians.  The  second  segment  of  Trial  1  shows  an  accumulation  that  is 
significantly  steeper  than  the  other  trials  especially  compared  to  the  second  segment  of 
Trial  1  for  the  Armed  Civilians.  Although  there  is  not  a  significant  difference  between 
the  trials  of  the  Mechanized  IDs,  the  difference  between  the  two  target  groups  is 
interesting. 


Accumulated  IDs  vs  Time 
Mechanized  Targets  -48  Hours 
(Proportion  of  Total) 


Time 


0.0 


-o-  Trial  1 

-□-Trial  2a 

-x-  Trial  3a 

-x-  T rial  4a 

-o-  Trial  4b 

-o-  Series3 

_ 

Figure  9.  Proportion  of  Total  Accumulated  IDs  vs.  Time  for  Mechanized  Targets 
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Summary 

This  chapter  detailed  various  attempts  to  distill  useful  information  from  the  vast 
data  set  created  in  the  URBAN  RESOLVE  exercise  despite  significant  limitations  due  to 
the  experimental  design  and  lack  of  replications.  Non-parametric  statistical  techniques 
were  used  to  calculate  95%  confidence  intervals  for  the  median  TTI  for  all  target  types 
and  for  each  target  group.  Analysis  of  all  Anned  Civilian  and  Mechanized  target  group 
IDs  showed  that  Trials  2a  and  3a  had  non-zero  TTIs  with  95%  confidence.  Analysis  of 
only  the  identifications  requiring  a  second-look  (TTI  >  0)  showed  some  interesting, 
though  inconclusive,  trends  that  run  counter  to  what  would  be  anticipated.  This  chapter 
also  discussed  the  analysis  of  TTI  data  relative  to  the  initial  detecting  platform  (UAV  vs. 
OAV).  Finally,  this  chapter  discussed  the  analysis  of  the  accumulation  rate  of 
identification  versus  time  for  the  Armed  Civilian  and  Mechanized  target  groups.  Some 
interesting  trends  were  noted  which  may  indicate  possible  areas  for  deeper  analysis. 
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V.  Conclusions 


Chapter  Overview 

This  chapter  will  provide  a  summary  discussion  of  this  research  effort.  First,  it 
will  discuss  the  results  of  the  analysis  performed  during  this  research  effort.  Next  will  be 
a  brief  discussion  of  the  obstacles  encountered  during  the  course  of  this  research.  Finally, 
it  will  suggest  areas  for  further  research  in  this  topic  area. 

Analysis  Results 

This  research  identified  two  notable  trends  in  the  URBAN  RESOLVE  data.  First, 
despite  the  “learning”  that  took  place  during  Trial  1,  the  TTI  for  all  target  types  was  zero 
minutes  for  all  of  the  trials.  When  looking  at  individual  target  groups,  the  Armed 
Civilians  were  the  only  target  group  with  a  Trial  1  TTI  that  stood  out  as  different  from 
the  other  target  groups.  When  examining  only  the  IDs  with  TTI  >  0,  the  Armed  Civilians 
in  Trial  1  seemed  to  possess  a  noticeably  longer  TTI.  This  result  is  to  be  expected  given 
the  “learning”  that  was  supposed  to  have  taken  place.  However,  when  examining  the 
TTIs  for  the  Mechanized  target  group,  Trial  1  was  reasonably  similar  compared  to  the 
other  trials.  Finally,  analysis  of  the  accumulated  IDs  versus  time  showed  a  similar  result 
where  fewer  Armed  Civilians  were  identified  at  a  slower  pace  in  Trial  1  compared  to  the 
other  trials.  On  the  other  hand,  the  exact  opposite  was  true  for  the  Mechanized  target 
group  where  many  more  targets  were  identified  at  a  much  faster  rate  than  the  other  trials. 

Second,  although  it  is  difficult  to  make  causal  comparisons  relative  to  the  design 
parameters  and  results,  Trials  4a  and  4b  provide  the  best  chance  to  observe  the  difference 
between  what  can  be  considered  the  best  case  scenario  (full  inventory  /  full  signature)  and 
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worst  case  scenario  (reduced  inventory  /  reduced  signature)  with  no  other  differences  in 
design.  However,  what  was  observed  most  often  was  at  best  a  complete  lack  of 
differentiation  between  the  two  trials,  and  at  worst,  results  that  appeared  completely 
counter  intuitive  to  the  expected  results.  The  initial  URBAN  RESOLVE  report  noted  that 
Trials  4a  and  4b  helped  demonstrate  the  importance  of  Tags  when  compared  to  the  other 
trials. 

The  two  trends  mentioned  above  raise  questions  that  bare  further  study: 

•  Why  are  the  Armed  Civilians  noticeably  different  from  the  other  target 
groups? 

•  Why  does  the  Mechanized  target  group  exhibit  different  behavior  from  the 
Armed  Civilians? 

•  Why  are  the  results  from  Trials  4a  and  4b  so  similar  in  some  cases  and 
counter-intuitive  in  others? 

Obstacles 

During  the  initial  phases  of  this  research,  the  intent  was  to  investigate  possible 
applications  of  JSAF  in  simulating  the  Air  and  Space  Operations  Center  (AOC)  and 
possible  applications  related  to  modeling  of  the  time  sensitive  targeting  (TST)  kill-chain. 
The  goal  was  to  generate  an  appropriate  environment  and  scenario  in  JSAF  and  to  use 
that  environment  to  investigate  the  ability  to  simulate  the  Combat  Operations  TCT  cell  of 
an  AOC.  However,  it  became  apparent  that  such  an  effort  was  too  large  an  undertaking 
for  the  time  and  resources  available. 

As  research  progressed,  and  the  existence  of  JSAF’s  use  by  other  agencies 
became  known,  the  hope  was  to  build  on  their  research  and  possibly  modify  their 
scenarios.  Attempts  were  made  to  acquire  some  portion  of  the  URBAN  RESOLVE 
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scenario  in  the  hope  that  it  could  be  used  to  perfonn  simulations  locally.  However,  even 
that  proved  impossible.  A  JSAF  scenario  cannot  be  easily  broken  down  into  pieces  for 
independent  use.  The  very  strengths  of  JSAF  that  make  it  such  a  powerful  tool,  its  ability 
to  operate  in  real-time  with  HITL  participants  and  a  large  numbers  of  autonomous 
entities  generated  through  multiple  distributed  computing  clusters,  makes  its  utility  trivial 
when  operated  on  a  single  workstation.  A  project  that  seemed  ambitious  at  first  quickly 
became  impossible  as  time  marched  on. 

Ultimately,  the  scope  and  scale  of  the  URBAN  RESOLVE  exercise  presented  a 
possible  foothold  for  research.  The  sheer  size  of  the  dataset  promised  to  be  a  treasure 
trove  for  analysis,  however  very  size  of  the  dataset  made  it  difficult  to  obtain.  After 
nearly  five  weeks  of  unsuccessful  attempts  to  acquire  and  convert  a  four-hour  portion  of 
data  (8GB),  we  were  finally  able  to  access  the  entire  dataset  housed  on  the  J9  servers  via 
the  DREN. 

Recommendations  for  Future  Research 

Several  avenues  for  further  research  readily  present  themselves  in  relation  to  past 
and  future  incarnations  of  URBAN  RESOLVE.  Analysis  to  answer  the  questions  raised 
in  this  research  may  help  improve  the  design  of  future  URBAN  RESOLVE  events. 
Additional  research  using  more  sophisticated  data  mining  techniques  may  illuminate 
additional  trends  in  the  data  that  have  not  yet  been  discovered.  Another  area  for  future 
research  is  the  development  of  pattern  recognition  tools  to  cue  operators  towards  possible 
activity  of  interest  based  on  track  identity,  behavior,  and  proximity  to  other  tracks. 


48 


Finally,  multiple  replications  of  the  trials,  using  different  groups  of  operators,  may 
provide  a  deeper  analysis  and  expose  unseen  trends. 
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Appendix  A.  Analysis  Questions  and  Measures  of  Perfonnance 


Ql.  To  what  extent  can  SA/SU  be  developed  in  JUO 

Ml.l  Number  of  Critical  Command  Nodes  Identified 

a 

Terminal  points  of  Red  vehicle  paths 

b 

T erminal  points  of  Red  D1  paths 

c 

Red  Vehicle  congregation  points 

d 

Red  DI  congregation  points 

e 

Buildings  housing  several  (>  n)  Red  vehicles 

f 

Buildings  housing  many  (>  n)  Red  DI 

g 

Number  of  RF  transmissions  from  individual  bldgs 

h 

Number  of  RF  transmissions  from  specific  locations 

Ml. 2  Number  of  Critical  Weapon  Nodes  Identified 

a 

Terminal  points  of  Red  vehicle  paths  (see  Ml.l) 

b 

Terminal  points  of  Red  DI  paths  (see  Ml.l) 

c 

Acquisitions  of  Red  weapon  systems  (characteristics) 

d 

Acquisitions  of  Red  weapon  systems  (location) 

e 

Acquisitions  of  Red  WMD  systems  (characteristics) 

f 

Acquisitions  of  Red  WMD  systems  (location) 

Ml. 3  Number  of  Other  Critical  Nodes  (or  patterns)  Identified 

a 

Terminal  points  of  Red  vehicle  paths  (Ml.l) 

b 

Terminal  points  of  Red  DI  paths  (Ml.l) 

c 

Red  Vehicle  congregation  points  (Ml.l) 

d 

Red  DI  congregation  points  (Ml.l) 

e 

Vehicle  relative  positions  while  in  transit  (signature  formations) 

f 

DI  relative  positions  while  in  transit  (signature  formations) 

g 

Number  of  features  detected 

h 

Number  of  features  recognized 

I 

Number  of  features  identified 

j 

Locations  of  detected  features 

k 

Locations  of  recognized  features  (by  category) 

1 

Locations  of  identified  features  (by  category) 
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Ml. 4  Anticipated  Red  Activities 

a 

Number  of  tracks  of  duration  >  n  minutes  [n=5,  10,  15,  ...] 

b 

Number  of  tracks  in  which  targets  are  correctly  identified  for  >  n  minutes 

c 

Number  of  tracks  in  which  targets  are  correctly  recognized  for  >  n  minutes 

d 

Number  of  tracks  in  which  targets  are  correctly  classified  for  >  n  minutes 

e 

Number  of  false  targets  tracked  for  >  n  minutes 

f 

Number  of  tracks  in  which  targets  are  incorrectly  classified  for  >  n  minutes 

g 

Number  of  tracks  in  which  targets  are  incorrectly  recognized  for  >  n  minutes 

h 

Number  of  tracks  in  which  targets  are  incorrectly  identified  for  >  n  minutes 

i 

Duration  of  tracks  for  all  categories  above 

j 

Number  of  features  detected  through  "Change  Detection" 

k 

Number  of  features  recognized  through  "Change  Detection" 

1 

Number  of  features  identified  through  "Change  Detection" 

m 

Locations  of  features  detected  through  "Change  Detection" 

n 

Locations  of  features  recognized  through  "Change  Detection" 

0 

Locations  of  features  identified  through  "Change  Detection" 

Q2.  How  effectively  are  sensor  data  used  to  detect  and  correlate  targets? 

M2.1.  Track  Profile  and  Utility 

a 

Number  of  tracks  of  duration  >  n  minutes;  n  =  5,  10,  ... 

b 

Track  identifiers  (trks  >  n  min) 

c 

Target  ids  (trks  >  n  min) 

d 

Major  and  minor  axes  of  position  estimates  (trks  >  n  min) 

e 

True  target  type  (trks  >  n  min) 

f 

Perceived  target  type  (trks  >  n  min) 

g 

Sensors  contributing  to  establishing/maintaining  the  track 

h 

Platforms  contributing  to  establishing/maintaining  the  track 

I 

Sensor-Platform  combinations  (e.g.,  MW1R/  NFOV  -  UAV  3)  contributing  to  establishing/ 
maintaining  the  track 

M2. 2.  Time  to  correlate  target  acquisitions 

a 

Number  requiring  >  n  minutes  from  detection  to  establishing  track;  n  =  5,  10,  ... 

b 

Number  requiring  >  n  minutes  from  detection  to  classification;  n  =  5,  10,  ... 

c 

Number  requiring  >  n  minutes  from  initial  acquisition  to  recognition;  n  =  5,  10,  ... 

d 

Number  requiring  >  n  minutes  from  initial  acquisition  to  identification;  n  =  5,  10,  ... 
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M2. 3  Effectiveness  of  Each  Sensor  System  (e.g.,  MWIR/  WFOV  ) 

a 

Population  of  targets  of  each  type  (function  of  time) 

b 

Bubble  charts  of  sensor  target  interactions  (number  of  interactions) 

c 

Number  of  each  type  target  detected  by  each  sensor  class  (unique  entities  detected) 

d 

Number  of  each  type  target  correctly  recognized  by  each  sensor  class 

e 

Number  of  each  type  target  correctly  identified  by  each  sensor  class 

f 

Number  of  each  type  target  incorrectly  recognized  by  each  sensor  class 

g 

Number  of  each  type  target  incorrectly  identified  by  each  sensor  class 

h 

"Failed  Detections"  by  Sensor  class  due  to  MDV 

I 

"Failed  Detections"  by  Sensor  class  due  to  [each  of]  bldg/foliage/vehicle  obscuration 

j 

"Failed  Detections"  per  Sensor  by  class  due  to  MDV 

k 

"Failed  Detections"  per  Sensor  by  class  due  to  [each  of]  bldg/foliage/vehicle  obscuration 

1 

Sensor  class  providing  initial  acquisition  of  each  target  (may  be  emission) 

m 

Sensor  class  providing  initial  correct  recognition  (possibly  none) 

n 

Sensor  class  providing  initial  correct  identification  (possibly  none) 

0 

Sensor  classes  (PLURAL)  responsible  for  establishing  track 

M2. 4  Effectiveness  of  Each  Sensor-Platform  Class  (e.g.,  MW1R/WFOV  -  UAV3) 

a 

Population  of  targets  of  each  type  (function  of  time)  [see  M.2.3] 

b 

Number  of  each  type  target  detected  by  each  sensor-platform  class 

c 

Number  of  each  type  target  correctly  recognized  by  each  sensor-platform  class 

d 

Number  of  each  type  target  correctly  identified  by  each  sensor-platform  class 

e 

Number  of  each  type  target  incorrectly  recognized  by  each  sensor-platform  class 

f 

Number  of  each  type  target  incorrectly  identified  by  each  sensor-platform  class 

g 

"Failed  Detections"  by  Sensor-Platform  class  due  to  MDV 

h 

"Failed  Detections"  by  Sensor-Platform  class  due  to  [each  of]  bldg/foliage/vehicle  obscuration 

i 

"Failed  Detections"  per  Sensor-Platform  by  class  due  to  MDV 

i 

"Failed  Detections"  per  Sensor-Platform  by  class  due  to  [each  of]  bldg/foliage  obscuration 

k 

Sensor-Platform  class  providing  initial  acquisition  of  each  target 

1 

Sensor-Platform  class  providing  initial  recognition 

m 

Sensor-Platform  class  providing  initial  identification 

n 

Sensor-Platform  classes  responsible  for  establishing  track 

0 

Sensor-Platform  classes  responsible  for  update  that  "correctly  identifies"  track 
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M2. 5  Effectiveness  of  Each  Platform  Class  (e.g.,  UAV3) 

a 

Population  of  targets  of  each  type  (function  of  time)  [see  M.2.3] 

b 

Number  of  each  type  target  detected  by  each  platform  class  (unique  entities) 

c 

Number  of  each  type  target  correctly  recognized  by  each  platform  class 

d 

Number  of  each  type  target  correctly  identified  by  each  platform  class 

e 

Number  of  each  type  target  incorrectly  recognized  by  each  platform  class 

f 

Number  of  each  type  target  incorrectly  identified  by  each  platform  class 

g 

"Failed  Detections"  by  Platform  class  due  to  MDV 

h 

"Failed  Detections"  by  Platform  class  due  to  [each  of]  bldg/foliage/vehicle  obscuration 

i 

"Failed  Detections"  per  Platform  by  class  due  to  MDV 

j 

"Failed  Detections"  per  Platform  by  class  due  to  [each  of]  bldg/foliage/vehicle  obscuration 

k 

Platform  class  providing  initial  acquisition  of  each  target 

1 

Platform  class  providing  initial  recognition 

m 

Platform  class  providing  initial  identification 

n 

Platform  classes  responsible  for  establishing  track 

M2. 6  Sensor  platform  tasking  and  retasking  responsiveness 

a 

Time  of  each  sensor  request,  by  type  of  sensor  and  platform 

b 

Type  of  request  (automated  retasking,  operator  decisions,  etc.) 

c 

Time  request  is  received  by  controlling  agent  (sensor  type  and  platform) 

d 

Time  task  or  retask  begins  (sensor  type  and  platform) 

e 

Time  of  arrival  of  retasked  platform  in  target  area 

f 

Length  of  retasking  chain  [how  many  platforms  are  retasked  serially,  parallel?] 

g 

Outcome  [failed  to  find  target]  [found  intended  target]  [found  something] 

h 

Time  [if  successful]  sensor  acquires  intended  target  (sensor  type  and  platform) 

M2. 7  Timing  and  events  in  target  prosecution  chain 

a 

Time  of  initial  detection 

b 

Type  of  sensor  performing  initial  detection  (see  M2. 3) 

c 

Type  of  platform  performing  initial  detection  (see  M2. 3) 

d 

Time  higher  level  sensor  is  tasked  to  query  detected  target 

e 

Type  of  sensor  tasked  for  higher  level  acquisition  (See  M2. 3) 

f 

Type  of  platform  tasked  for  higher  level  acquisition  (see  M2. 3) 

g 

Time  of  higher  level  acquisition(s) 

h 

Type  of  sensor  performing  higher  level  acquisition  (see  M2. 3) 

i 

Type  of  platform  performing  higher  level  acquisition  (see  M2. 3) 

,i 

Time  of  declaration  "target  of  interest" 

k 

Time  of  declaration  "target  is  recommended" 

l 

Operator  override  of  sensor-provided  track  identity  (time  and  imposed  type) 
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Q  3.  What  conditions  affect  Blue  capabilities? 

M3. 1  Impact  of  Terrain  and  Buildings  on  sensor  operations 

a 

RSS  Event  Codes  for  masking  (reasons  for  missed  detections) 

b 

Camouflage  State  of  missed  targets  in  sensor  window 

c 

Number  of  false  alarms  and  mis-acquisitions  in  "n"  minute  windows 

d 

Targets  never  detected  [type,  number,  location] 

e 

Locations  of  detected  targets  [downtown,  city  parks,  highway,  shanty  town,  etc] 

f 

Location  where  tracks  are  established  [downtown,  city  parks,  highway,  shanty  town,  etc] 

g 

Location  where  tracks  are  lost  [downtown,  city  parks,  highway,  shanty  town,  etc] 

h 

Time -position  history  of  gaps  in  tracks 

M3. 2  Temporal  effects  on  sensor  operations 

a 

RSS  Event  Codes  for  masking  (reasons  for  missed  detections)  by  time 

b 

Camouflage  State  of  missed  targets  in  sensor  window  by  time  (same  as  above?) 

c 

Number  of  false  alarms  and  mis-acquisitions  in  "n-minute"  windows 

d 

Time  tracks  are  established  [morning  rush  hour,  midday,  evening  rush,  nighttime,  etc] 

e 

Time  tracks  are  lost  [morning  rush  hour,  midday,  evening  rush,  nighttime,  etc] 

f 

Time  history  of  gaps  in  tracks 

M3. 3  Impact  of  Red  Counter  Measures  on  sensor  operations 

a 

RSS  Event  Codes  for  masking  (reasons  for  missed  detections) 

b 

Camouflage  State  of  missed  targets  in  sensor  window 

c 

Sensors  neutralized/compromised  by  deliberate  Red  action 

d 

Platforms  destroyed  by  Red  munitions 

M3. 4  Actions  most  favorable  to  Red 

a 

Red  formations  when  acquired  [small  group,  large  group,  individual  vehicle/DI] 

b 

Red  formations  when  not  acquired  [small  group,  large  group,  individual  vehicle/DI] 

c 

Red  Activity  when  acquired  [on  road  movement,  WMD  set  up,  static,  etc] 

d 

Red  Activity  when  not  acquired  [on  road  movement,  WMD  set  up,  static,  etc] 

e 

Location  where  acquired  [M3.1] 

f 

Location  where  not  acquired  [M3.1]  {where  Red  is  unlikely  to  be  detected] 

g 

Time  when  acquired  [M3. 2] 

h 

Time  when  not  acquired  [M3. 2]  {when  Red  is  unlikely  to  be  detected] 

M3. 5  Communications 

a 

Histogram  of  lengths  of  messages  sent  (bytes),  aggregated  and  by  transmitter  type 

b 

Histogram  of  lengths  of  messages  received  (bytes) 

c 

Number  of  messages  sent  as  a  function  of  time,  aggregated  and  by  transmitter  type 

d 

Number  of  messages  received  as  a  function  of  time 

e 

Number  of  messages  transmitted  but  not  received  due  to  no  LOS 

f 

Number  of  messages  transmitted  but  not  received  due  to  low  power  (SLMM) 

g 

Number  of  messages  transmitted  but  not  received  due  to  reasons  other  than  LOS  or  PWR 

h 

Histogram  of  number  of  lost  messages  by  reason  for  loss. 
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Appendix  B.  Additional  Analysis  Results 
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Mean 

STD 
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600 
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04b 
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2.0 
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Other  Target  Types  All  IDs 
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Total 

Detections 
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IDs 
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ID 

Time 

Mean 

STD 

Median 

■ 

Lr 
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ID 
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01 

8398 
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91% 
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0.2 

1.1 

0.0 

747 

0.0 

0.0 
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0.0 

0.0 
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Artillery  Target  Types,  All  IDs 


1 
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Detections 
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Max 
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Time 

Mean 

STD 
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Lr 

Ur 
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Artillery 

01 
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25.1 
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Air  Defense  Target  types,  All  IDs 
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